GBO Preliminary Research Proposal

Gray and Szalay [1] documented the data avalanche problem in the sciences in which improvements in physical instruments and better data pipelines lead to an exponential growth in data size. Paralleling this exponential trend is the accumulation of data at multiple, autonomous data sources. Exploring the resulting massive, widely-distributed data is of immense scientific value: as Gray and Szalay observe, the number of scientific discoveries increase polynomially with the number of participating data sources [2]. Federated databases is an attractive solution for the management and sharing of scientific data that are geographically distributed. Increasingly, Science discoveries are made by scanning large portions of the data to find correlations, mine data, extract features, and compute joins across distributed data sources [3]. These “needle in a haystack” queries are long running and data intensive so that query throughput limits performance. In order to facilitate data exploration, various scientific disciplines have built federations of databases. Federations allow data, which are too large to be widely replicated or stored at a single site, to be managed independently. Examples include SkyQuery [4], Genbank [5], and EcoliHub [6]. To ensure high job throughput and prevent starvation of traditional workloads in a federated environment, we propose new query processing disciplines. Scientific database federations built at a global scale render many goals of distributed query processing obsolete. Specifically, queries are I/O intensive and require non-indexed scans of multi-terabyte tables, which may take several hours to complete [3]. Data size and geography also dictate that transmitting data takes large amounts of time and has a profound impact on query performance. Thus, workloads are not latency sensitive due to the large data sizes (accessing data at each site takes tens of seconds at best). Our goal is maximizing query throughput in the federation through query scheduling techniques that incorporate network structure (exploit high capacity network paths) and account for data access requirements (maximize data sharing among queries). A guiding principle of our work is that rather than choosing the optimal plan for each query, we choose plans that penalize other concurrent queries minimally and improve overall query throughput. For example, algorithms that minimize completion time over-utilize the network by consuming all available resources to achieve a locally optimal plan [7]. Thus, we limit the amount of parallelism in query schedules to avoid multiple data transfers across large geographies. We also delay Web Mediator WHERE XMATCH(o, t, p) < 3.5 and REGION(’circle 181.3 −0.76 6.5’) SELECT ... FROM SDSS o, TWOMASS t, USNOB p